17 research outputs found
RobustCLEVR: A Benchmark and Framework for Evaluating Robustness in Object-centric Learning
Object-centric representation learning offers the potential to overcome
limitations of image-level representations by explicitly parsing image scenes
into their constituent components. While image-level representations typically
lack robustness to natural image corruptions, the robustness of object-centric
methods remains largely untested. To address this gap, we present the
RobustCLEVR benchmark dataset and evaluation framework. Our framework takes a
novel approach to evaluating robustness by enabling the specification of causal
dependencies in the image generation process grounded in expert knowledge and
capable of producing a wide range of image corruptions unattainable in existing
robustness evaluations. Using our framework, we define several causal models of
the image corruption process which explicitly encode assumptions about the
causal relationships and distributions of each corruption type. We generate
dataset variants for each causal model on which we evaluate state-of-the-art
object-centric methods. Overall, we find that object-centric methods are not
inherently robust to image corruptions. Our causal evaluation approach exposes
model sensitivities not observed using conventional evaluation processes,
yielding greater insight into robustness differences across algorithms. Lastly,
while conventional robustness evaluations view corruptions as
out-of-distribution, we use our causal framework to show that even training on
in-distribution image corruptions does not guarantee increased model
robustness. This work provides a step towards more concrete and substantiated
understanding of model performance and deterioration under complex corruption
processes of the real-world
Context-Adaptive Deep Neural Networks via Bridge-Mode Connectivity
The deployment of machine learning models in safety-critical applications
comes with the expectation that such models will perform well over a range of
contexts (e.g., a vision model for classifying street signs should work in
rural, city, and highway settings under varying lighting/weather conditions).
However, these one-size-fits-all models are typically optimized for average
case performance, encouraging them to achieve high performance in nominal
conditions but exposing them to unexpected behavior in challenging or rare
contexts. To address this concern, we develop a new method for training
context-dependent models. We extend Bridge-Mode Connectivity (BMC) (Garipov et
al., 2018) to train an infinite ensemble of models over a continuous measure of
context such that we can sample model parameters specifically tuned to the
corresponding evaluation context. We explore the definition of context in image
classification tasks through multiple lenses including changes in the risk
profile, long-tail image statistics/appearance, and context-dependent
distribution shift. We develop novel extensions of the BMC optimization for
each of these cases and our experiments demonstrate that model performance can
be successfully tuned to context in each scenario.Comment: Accepted to the NeurIPS 2022 ML Safety Worksho
A Systematic Review of Robustness in Deep Learning for Computer Vision: Mind the gap?
Deep neural networks for computer vision are deployed in increasingly
safety-critical and socially-impactful applications, motivating the need to
close the gap in model performance under varied, naturally occurring imaging
conditions. Robustness, ambiguously used in multiple contexts including
adversarial machine learning, refers here to preserving model performance under
naturally-induced image corruptions or alterations.
We perform a systematic review to identify, analyze, and summarize current
definitions and progress towards non-adversarial robustness in deep learning
for computer vision. We find this area of research has received
disproportionately less attention relative to adversarial machine learning, yet
a significant robustness gap exists that manifests in performance degradation
similar in magnitude to adversarial conditions.
Toward developing a more transparent definition of robustness, we provide a
conceptual framework based on a structural causal model of the data generating
process and interpret non-adversarial robustness as pertaining to a model's
behavior on corrupted images corresponding to low-probability samples from the
unaltered data distribution. We identify key architecture-, data augmentation-,
and optimization tactics for improving neural network robustness. This
robustness perspective reveals that common practices in the literature
correspond to causal concepts. We offer perspectives on how future research may
mind this evident and significant non-adversarial robustness gap
MyD88 Primes Macrophages for Full-Scale Activation by Interferon-γ yet Mediates Few Responses to Mycobacterium tuberculosis
Macrophages are activated from a resting state by a combination of cytokines and microbial products. Microbes are often sensed through Toll-like receptors signaling through MyD88. We used large-scale microarrays in multiple replicate experiments followed by stringent statistical analysis to compare gene expression in wild-type (WT) and MyD88−/− macrophages. We confirmed key results by quantitative reverse transcription polymerase chain reaction, Western blot, and enzyme-linked immunosorbent assay. Surprisingly, many genes, such as inducible nitric oxide synthase, IRG-1, IP-10, MIG, RANTES, and interleukin 6 were induced by interferon (IFN)-γ from 5- to 100-fold less extensively in MyD88−/− macrophages than in WT macrophages. Thus, widespread, full-scale activation of macrophages by IFN-γ requires MyD88. Analysis of the mechanism revealed that MyD88 mediates a process of self-priming by which resting macrophages produce a low level of tumor necrosis factor. This and other factors lead to basal activation of nuclear factor κB, which synergizes with IFN-γ for gene induction. In contrast, infection by live, virulent Mycobacterium tuberculosis (Mtb) activated macrophages largely through MyD88-independent pathways, and macrophages did not need MyD88 to kill Mtb in vitro. Thus, MyD88 plays a dynamic role in resting macrophages that supports IFN-γ–dependent activation, whereas macrophages can respond to a complex microbial stimulus, the tubercle bacillus, chiefly by other routes
Exploiting Large Neuroimaging Datasets to Create Connectome-Constrained Approaches for more Robust, Efficient, and Adaptable Artificial Intelligence
Despite the progress in deep learning networks, efficient learning at the
edge (enabling adaptable, low-complexity machine learning solutions) remains a
critical need for defense and commercial applications. We envision a pipeline
to utilize large neuroimaging datasets, including maps of the brain which
capture neuron and synapse connectivity, to improve machine learning
approaches. We have pursued different approaches within this pipeline
structure. First, as a demonstration of data-driven discovery, the team has
developed a technique for discovery of repeated subcircuits, or motifs. These
were incorporated into a neural architecture search approach to evolve network
architectures. Second, we have conducted analysis of the heading direction
circuit in the fruit fly, which performs fusion of visual and angular velocity
features, to explore augmenting existing computational models with new insight.
Our team discovered a novel pattern of connectivity, implemented a new model,
and demonstrated sensor fusion on a robotic platform. Third, the team analyzed
circuitry for memory formation in the fruit fly connectome, enabling the design
of a novel generative replay approach. Finally, the team has begun analysis of
connectivity in mammalian cortex to explore potential improvements to
transformer networks. These constraints increased network robustness on the
most challenging examples in the CIFAR-10-C computer vision robustness
benchmark task, while reducing learnable attention parameters by over an order
of magnitude. Taken together, these results demonstrate multiple potential
approaches to utilize insight from neural systems for developing robust and
efficient machine learning techniques.Comment: 11 pages, 4 figure
Comparative analysis of the transcriptome across distant species
The transcriptome is the readout of the genome. Identifying common features in it across distant species can reveal fundamental principles. To this end, the ENCODE and modENCODE consortia have generated large amounts of matched RNA-sequencing data for human, worm and fly. Uniform processing and comprehensive annotation of these data allow comparison across metazoan phyla, extending beyond earlier within-phylum transcriptome comparisons and revealing ancient, conserved features. Specifically, we discover co-expression modules shared across animals, many of which are enriched in developmental genes. Moreover, we use expression patterns to align the stages in worm and fly development and find a novel pairing between worm embryo and fly pupae, in addition to the embryo-to-embryo and larvae-to-larvae pairings. Furthermore, we find that the extent of non-canonical, non-coding transcription is similar in each organism, per base pair. Finally, we find in all three organisms that the gene-expression levels, both coding and non-coding, can be quantitatively predicted from chromatin features at the promoter using a 'universal model' based on a single set of organism-independent parameters